Occupant facing vehicle display

ABSTRACT

Aspects of the present disclosure relate to a vehicle for maneuvering an occupant of the vehicle to a destination autonomously as well as providing information about the vehicle and the vehicle&#39;s environment for display to the occupant.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/165,283, filed Feb. 8, 2019, which is a continuation of U.S.patent application Ser. No. 15/089,736, filed on Apr. 4, 2016, nowissued as U.S. Pat. No. 10,140,870, which claims the benefit of thefiling date of U.S. Provisional Patent Application No. 62/234,814 filedSep. 30, 2015, the entire disclosures of which are incorporated byreference herein. The present application is related to application Ser.Nos. 15/089,708, 15/089,716, and 15/089,776, filed Apr. 4, 2016,entitled Occupant Facing Vehicle Display, and the entire disclosures ofwhich are incorporated by reference herein.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or items fromone location to another. Such vehicles may operate in a fully autonomousmode where passengers may provide some initial input, such as adestination, and the vehicle maneuvers itself to that destination.

A key component of an autonomous vehicle is the perception system, whichallows the vehicle to perceive and interpret its surroundings during atrip. When the autonomous system is engaged, the system will makevarious decisions during the trip, for example, speed up, slow down,stop, etc. The operator may be unaware of the calculations or“reasoning” behind why the autonomous vehicle is taking some particularaction. In order to feel safe and confident, the operator may want toknow what the vehicle is planning to do in the immediate future.

BRIEF SUMMARY

One aspect of the disclosure provides a method. The method includesreceiving, by one or more processors, sensor data from one or moresensor systems configured to detect objects in an external environmentof a vehicle; identifying, by the one or more processors, an object inthe vehicle's environment using the sensor data; determining, by the oneor more processors, that the vehicle must yield to the identified objectin order to safely maneuver the vehicle to a destination; generating, bythe one or more processors, a representation of the object for a scenedisplay depicting the external environment of the vehicle, therepresentation of the object including a feature having a first color;generating, by the one or more processors, an intent indicator thatindicates the determination that the vehicle must yield to theidentified object for an intent display depicting the intent of thevehicle with respect to the external environment of the vehicle, theintent indicator including a feature having the first color; andproviding, by the one or more processors, the representation of theobject and intent indicator for display to an occupant of the vehicle inorder to indicate to the occupant a relationship between therepresentation of the object and the intent indicator through the firstcolor.

In one example, the representation of the object is provided for displayin a first area of an electronic display corresponding to the scenedisplay, the intent indicator is provided for display in a second areaof an electronic display distinct from the scene display. In anotherexample, the representation of the object is provided for display in afirst area of an electronic display corresponding to the scene displayand the intent indicator is provided for display as an overlay on thefirst area of the electronic display over the scene display. In anotherexample, an area within the intent indicator which overlays a portionthe scene display includes a blur effect for the portion in order todistinguish the intent indicator from the scene display. In anotherexample, the method also includes generating a three-dimensionalbackground scene using detailed map information corresponding to theexternal environment of the vehicle, and the background scene isprovided as a background for the representation of the object and theintent indicator is provided as a two-dimensional image configured to beoverlaid onto the three-dimensional background scene. In anotherexample, the feature of the representation of the object is a firstring, and the feature of the intent indicator is a second ring. Inanother example, the feature of the representation of the object is afirst ring drawn around a depiction of sensor data corresponding to theobject and the feature of the intent indicator is a second ring drawnaround a depiction of intent of one or more computing devices of thevehicle. In another example, the feature of the representation of theobject is an abstract depiction of the object and the feature of theintent indicator is a ring drawn around a depiction of intent of one ormore computing devices of the vehicle.

A further aspect of the disclosure provides a method. The methodincludes receiving, by one or more processors, sensor data from a sensorconfigured to detect objects in an external environment of a vehicle;identifying, by the one or more processors, a first object and a secondobject in the vehicle's environment from the sensor data; determining,by the one or more processors, that the first object is relevant tosafely maneuvering the vehicle to a destination; determining, by the oneor more processors, that the second object is not currently relevant tosafely maneuvering the vehicle to the destination; generating, by theone or more processors, a representation of the first object for a scenedisplay depicting the external environment of the vehicle, therepresentation of the first object including a feature having a firstcolor scheme; generating, by the one or more processors, arepresentation of the second object for the scene display, therepresentation of the second object having a second color scheme,wherein the first color scheme is configured to stand out from thesecond color scheme; generating, by the one or more processors, athree-dimensional background scene in the second color scheme usingdetailed map information corresponding to the external environment ofthe vehicle; and providing, by the one or more processors, therepresentation of the first object, the representation of the secondobject, and the background scene for display to an occupant of thevehicle in order to indicate to the occupant that first object isrelevant to safely maneuvering the vehicle to a destination and that thesecond object is not currently relevant to safely maneuvering thevehicle to the destination.

In one example, the background scene includes lane lines in the secondcolor scheme, and the representation of the first object and therepresentation of the second object both represent vehicles. In anotherexample, the background scene includes static objects detected thevehicle's external environment and included with the detailed mapinformation. In another example, the sensor data indicates that both thefirst object and the second object are of a same type. In anotherexample, the method also includes, after providing the firstrepresentation for display, determining that the first object should nolonger be shown as relevant to safely maneuvering the vehicle to thedestination; generating, based on the determination that the firstobject should no longer be shown as relevant, a new representation ofthe first object in the second color scheme; and providing the newrepresentation as a replacement for the representation of the firstobject in order to indicate to the occupant that the first object is nolonger relevant to safely maneuvering the vehicle to the destination. Inanother example, the new representation is provided such that therepresentation of the first object will appear to fade into thebackground scene by being replaced with the new representation. Inanother example, after providing the second representation for display,determining that the second object is relevant to safely maneuvering thevehicle to the destination; generating a new representation of thesecond object in the second color scheme; and providing the newrepresentation as a replacement for the representation of the secondobject in order to indicate to the occupant that the first second isrelevant to safely maneuvering the vehicle to the destination. Inanother example, the new representation is provided such that the newrepresentation of the first object will appear to fade in from thebackground scene when replacing the representation of the first object.In another example, determining that the first object is relevant tosafely maneuvering the vehicle to the destination is based on adetermination that the vehicle must take a specific action in order toavoid the first object for a predetermined period of time into thefuture. In this example, determining that the second object is notcurrently relevant to safely maneuvering the vehicle to the destinationis based on a determination that the vehicle need not take any specificaction in order to avoid the second object for a predetermined period oftime into the future. In another example, the method also includesdetermining a route to a destination; generating a path for display inthe scene display representing the route; providing the route fordisplay in a third color scheme different from the second color schemein order to distinguish the path from the background scene; and afterproviding the route for display, when the vehicle has stopped to avoidan object, providing the route for display in the second color scheme inorder to indicate to the occupant that the vehicle is temporarilystopped but will shortly resume the route.

Another aspect of the disclosure provides a method. The method includesreceiving, by one or more processors, sensor data from a sensorconfigured to detect objects in an external environment of a vehicle;identifying, by the one or more processors, a first object of a firsttype and a second object of a second type in the vehicle's environmentfrom the sensor data, the first type and the second type beingdifferent; generating, by the one or more processors, a representationof the first object for a scene display depicting the externalenvironment of the vehicle based on the first type, wherein therepresentation of the first object includes a display of at least aportion of the sensor data corresponding to the object and indicates anintent of one or more of the vehicle's computers with regard to thefirst object; generating, by the one or more processors, arepresentation of the second object for the scene display based on thesecond type, wherein the representation of the second object is anabstract representation of an object and indicates an intent of the oneor more of the vehicle's computers with regard to the second object; andproviding, by the one or more processors, the representation of thefirst object and the representation of the second object for display onan electronic display to an occupant of the vehicle in order to indicateto the occupant that the first object and the second objects are ofdifferent types and that the first object is more vulnerable to animpact with the vehicle than the second object.

In one example, the representation of the first object includes a laserpoint cloud that provides the occupant with an indication of theparticular shape of the first object. In this example, therepresentation of the second object is a stylized cuboid that does notprovide the occupant with an indication of the particular shape of thefirst object. In another example, the first object is a pedestrian andthe second object is a vehicle. In another example, the first type isbicyclist and the second type is vehicle. In another example, the methodalso includes generating a three-dimensional background scene usingdetailed map information corresponding to the external environment ofthe vehicle; and providing the three-dimensional background scene fordisplay with the representation of the first object and therepresentation of the second object. In another example, the method alsoincludes identifying a third object of a third type in the vehicle'senvironment from the sensor data; generating a representation of thethird object that is a same representation as the representation of thesecond object; and providing the representation of the third object fordisplay with the representation of the first object and therepresentation of the second object. In this example, the third type isa same type as the second type. In addition, the second object and thefirst object are vehicles. Alternatively, the third type is a differenttype from the second type. In addition, the second object is a car andthe third object is a tractor trailer.

Yet another aspect of the disclosure provides a method. The methodincludes receiving, by one or more processors, sensor data from a sensorconfigured to detect objects in an external environment of a vehicle;generating, by the one or more processors, a background scene fordisplay corresponding to an area of the external environment of thevehicle; identifying, by the one or more processors, a set of objects inthe area from the sensor data, the objects of the set of objectscorresponding to one or more of a vehicle, a pedestrian, and abicyclist; filtering, by the one or more processors, the set of objectsbased on a set of heuristics; generating, by the one or more processors,a representation of each object of the filtered second set of objects;and providing, by the one or more processors, any generatedrepresentations and the background scene for display to an occupant ofthe vehicle on an internal electronic display of the vehicle, whereinthe filtering reduces clutter on the internal electronic display.

In one example, the heuristics include a vulnerability of the object. Inthis example, the vulnerability of a pedestrian gives an objectcorresponding to a pedestrian a higher priority to be included in thefiltered set of objects than an object corresponding to a vehicle. Inaddition or alternatively, the vulnerability of a bicyclist gives anobject corresponding to a bicyclist a higher priority to be included inthe filtered set of objects than an object corresponding to a vehicle.In addition or alternatively, the vulnerability of a pedestrian gives anobject corresponding to a pedestrian a higher priority to be included inthe filtered set of objects than an object corresponding to a bicyclist.In another example, the filtering is based on a category of action ofeach object of the set of objects. In another example, the filtering isbased on whether an object of the set of objects is in a same lane asthe vehicle. In another example, the filtering is based on whether anobject of the set of objects is merging into a same lane as the vehicle.In another example, the filtering is based on whether an object of theset of objects is turning into in a same lane as the vehicle. In anotherexample, the filtering is based on whether the vehicle needs to take anaction to avoid an object of the set of objects.

The disclosure also provides for non-transitory, tangible computerreadable medium on which instructions are stored, the instructions whenexecuted by one or more processors cause the one or more processors toperform any of the methods discussed above and herein. In addition, thedisclosure also provides for systems including one or more computingdevices having one or more processors configured to perform any of themethods discussed above and herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a functional diagram of an example vehicle in accordance withaspects of the disclosure.

FIG. 2 is an example representation of detailed map information inaccordance with aspects of the disclosure.

FIGS. 3A-3D are example external views of a vehicle in accordance withaspects of the disclosure.

FIG. 4 is an example internal view of a vehicle in accordance withaspects of the disclosure.

FIG. 5 is an example of a console of a vehicle in accordance withaspects of the disclosure.

FIG. 6 is a functional diagram of an example system in accordance withaspects of the disclosure.

FIG. 7 is a pictorial diagram of the system of FIG. 6 in accordance withaspects of the disclosure.

FIG. 8 is an example of an intersection in accordance with aspects ofthe disclosure.

FIG. 9 is an example of a display in accordance with aspects of thedisclosure.

FIG. 10 is example representations of objects in accordance with aspectsof the disclosure.

FIG. 11 is further example representations of objects in accordance withaspects of the disclosure.

FIG. 12 is additional example representations of objects in accordancewith aspects of the disclosure.

FIG. 13 is example representations of objects in accordance with aspectsof the disclosure.

FIG. 14 is an example flow diagram in accordance with aspects of thedisclosure.

FIG. 15 is an example flow diagram in accordance with aspects of thedisclosure.

FIG. 16 is an example flow diagram in accordance with aspects of thedisclosure.

FIG. 17 is an example flow diagram in accordance with aspects of thedisclosure.

DETAILED DESCRIPTION

The technology relates to a vehicle for maneuvering a passenger to adestination, for example taking a trip, autonomously. In order toprovide a safe, enjoyable and relaxing experience for passengers, it isimportant to be able to relate to providing information to an occupantof a vehicle about the vehicle's surrounding environment as well asabout how the vehicle's computers are reacting to those surroundings. Asan example, one or more in-vehicle displays may provide informationabout objects in a vehicle's surroundings. How and which objects aredepicted can be used in order to quickly and easily indicate informationabout the objects to an occupant of the vehicle. For instance, differenttypes or categories of objects, such as pedestrians, vehicles, bicycles,static objects, moving objects, etc., can be displayed with differentimage treatments and colors to indicate different types of informationto an occupant.

When the vehicle operates in a semiautonomous or fully autonomous mode,the vehicle's computers may not only detect these objects, but alsorespond or react to them. Thus, the in-vehicle display can displayinformation about how the vehicle's computers plan to react to thoseobjects, or rather, the vehicle's intent. In this regard, the in-vehicledisplay can include a scene display for displaying a one or morerepresentative objects and a scene corresponding to an area around thevehicle and an intent display for displaying intent of the vehicle withrespect to one or more of these objects.

The scene and intent displays can be arranged in various ways. Forinstance, the scene display and intent display may be displayed on thesame or different electronic displays of the vehicle. The intent displaymay therefore actually include a sub-area of the scene display such as aside or corner area of the scene display such that the intent displayappears to overlap or is overlaid or superimposed on the scene display.Similarly, the intent display and the scene display may be depicted asdiscrete areas of an electronic display such that there is no overlap oroverlay.

In addition, the scene display and intent display may be distinguishedbased on how the features in these displays are depicted. As an example,the scene of the scene display may be a three-dimensional (3D) imagehaving a depth while the features of the intent display may appear to beflat or two-dimensional (2D). Alternatively, both the scene and intentdisplays may appear to be in 2D or 3D.

The relationship between a representative object and the vehicle'sintent relative to an object may be presented using colors. As anexample, a pedestrian representation shown in the scene display may havea yellow ring or other indicator, while corresponding intent to yield(or simply an acknowledgement that an object has been identified as apedestrian) may be displayed with a corresponding yellow ring in theintent display. Similarly, a representation of a vehicle shown in thescene display with a purple ring or indicator and a corresponding intentto yield to the vehicle may be displayed in the intent display with acorresponding purple ring or indicator. Also, different types of objectscan be shown with the same color in the scene display when the sameintent is shown in the intent display (i.e. yielding to a bicyclist anda vehicle).

In many examples, the vehicle's perception system will detectsignificantly more objects in the vehicle's external environment thanare useful or helpful to display to a passenger. Objects that areidentified must therefore be filtered. Filtering may be achieved, forexample, using various heuristics related to the vulnerability of anobject (pedestrians and bicyclists having higher priority than otherobjects unless those other objects are doing something elseinteresting), the category of action for the object (moving in the samelane, merging, turning into the lane, etc.), whether the vehicle needsto take an action to avoid the object (change lanes, slow down, turn thewheels, etc.), distance from the vehicle, speed of the object, etc. Inaddition, certain road features, such as construction zones or trafficsignal lights, may be omitted from the scene display, but the intent forthose features (cautious or slower than usual driving and/or stopping)may still be indicated in the intent display. By doing so, this willreduce the amount of “clutter” in the object display.

In some instances, representations of different types or categories ofobjects can be shown in the scene display using different imagetreatments in order to quickly and easily indicate information about theobjects to a passenger in the vehicle. For example, static objects orbackground features, such as lane lines, crosswalks, or other roadfeatures, may be displayed in one color scheme (i.e. all blues orgrays), while objects that more important, such as vehicles,pedestrians, or bicyclists may be shown in different colors in order todistinguish them from the background features. For instance, movingobjects to which the vehicle is actually reacting (i.e. changing itsbehavior to slow down, yield, etc.) may be distinguished from thebackground features using different colors. At the same time, movingobjects which do not affect the vehicle's behavior may also be shown inthe background color scheme.

In this regard, as objects become more or less important, theirprominence can be represented through changes in the color of therepresentations of those objects as shown in the scene display. As anexample, another vehicle in front of the vehicle may change from abrighter color to a duller color when the other vehicle has turned ontoa different roadway. Thus, the other vehicle will appear to “fade” intothe background of the scene display. In another example, another vehiclewhich moves into the vehicle's lane in front of the vehicle may changefrom a color of the background scheme to a brighter color as it becomesmore relevant to the vehicle. Similarly, the scene display may show arepresentation of a path of the vehicle which also fades to thebackground color scheme when the vehicle has come to a stop or istemporarily yielding to an object.

In addition, specific types of the moving objects may be represented inthe scene display in different ways. As an example, more vulnerabletypes of objects may be shown using laser point clouds. Point cloudsallow a passenger to potentially pick up on details of a pedestrian orbicyclist such as what direction they are facing or moving. Other lessvulnerable objects, such as vehicles, may be represented using a moreabstract representation such as a box, tray, or stylized cuboid. Thesemore generic representations in combination with the point cloudrepresentations given a passenger a good sense that the vehicle hasidentified different objects and can also distinguish between vulnerableand less vulnerable objects.

As shown in FIG. 1 , a vehicle 100 in accordance with one aspect of thedisclosure includes various components. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, busses, recreational vehicles,etc. The vehicle may have one or more computing devices, such ascomputing device 110 containing one or more processors 120, memory 130and other components typically present in general purpose computingdevices.

The memory 130 stores information accessible by the one or moreprocessors 120, including instructions 132 and data 134 that may beexecuted or otherwise used by the processor 120. The memory 130 may beof any type capable of storing information accessible by the processor,including a computing device-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the claimedsubject matter is not limited by any particular data structure, the datamay be stored in computing device registers, in a relational database asa table having a plurality of different fields and records, XMLdocuments or flat files. The data may also be formatted in any computingdevice-readable format.

The one or more processor 120 may be any conventional processors, suchas commercially available CPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 1 functionally illustrates theprocessor, memory, and other elements of computing device 110 as beingwithin the same block, it will be understood by those of ordinary skillin the art that the processor, computing device, or memory may actuallyinclude multiple processors, computing devices, or memories that may ormay not be stored within the same physical housing. As an example,internal electronic display 152 may be controlled by a dedicatedcomputing device having its own processor or central processing unit(CPU), memory, etc. which may interface with the computing device 110via a high-bandwidth or other network connection. In some examples, thiscomputing device may be a user interface computing device which cancommunicate with a user's client device. Similarly, the memory may be ahard drive or other storage media located in a housing different fromthat of computing device 110. Accordingly, references to a processor orcomputing device will be understood to include references to acollection of processors or computing devices or memories that may ormay not operate in parallel.

Computing device 110 may all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user input 150 (e.g., a mouse, keyboard,touch screen and/or microphone) and various electronic displays (e.g., amonitor having a screen or any other electrical device that is operableto display information). In this example, the vehicle includes aninternal electronic display 152 as well as one or more speakers 154 toprovide information or audio visual experiences. In this regard,internal electronic display 152 may be located within a cabin of vehicle100 and may be used by computing device 110 to provide information topassengers within the vehicle 100.

In one example, computing device 110 may be an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may capable of communicating with various components ofthe vehicle. For example, returning to FIG. 1 , computing device 110 maybe in communication with various systems of vehicle 100, such asdeceleration system 160 (for controlling braking of the vehicle),acceleration system 162 (for controlling acceleration of the vehicle),steering system 164 (for controlling the orientation of the wheels anddirection of the vehicle), signaling system 166 (for controlling turnsignals), navigation system 168 (for navigating the vehicle to alocation or around objects), positioning system 170 (for determining theposition of the vehicle), perception system 172 (for detecting objectsin the vehicle's environment), and power system 174 (for example, abattery and/or gas or diesel powered engine) in order to control themovement, speed, etc. of vehicle 100 in accordance with the instructions134 of memory 130 in an autonomous driving mode which does not requireor need continuous or periodic input from a passenger of the vehicle.Again, although these systems are shown as external to computing device110, in actuality, these systems may also be incorporated into computingdevice 110, again as an autonomous driving computing system forcontrolling vehicle 100.

The computing device 110 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevice 110 may navigate the vehicle to a destination location completelyautonomously using data from the detailed map information and navigationsystem 168. Computer 110 may use the positioning system 170 to determinethe vehicle's location and perception system 172 to detect and respondto objects when needed to reach the location safely. In order to do so,computer 110 may cause the vehicle to accelerate (e.g., by increasingfuel or other energy provided to the engine by acceleration system 162),decelerate (e.g., by decreasing the fuel supplied to the engine,changing gears, and/or by applying brakes by deceleration system 160),change direction (e.g., by turning the front or rear wheels of vehicle100 by steering system 164), and signal such changes (e.g., by lightingturn signals of signaling system 166). Thus, the acceleration system 162and deceleration system 162 may be a part of a drivetrain that includesvarious components between an engine of the vehicle and the wheels ofthe vehicle. Again, by controlling these systems, computer 110 may alsocontrol the drivetrain of the vehicle in order to maneuver the vehicleautonomously.

As an example, computing device 110 may interact with decelerationsystem 160 and acceleration system 162 in order to control the speed ofthe vehicle. Similarly, steering system 164 may be used by computingdevice 110 in order to control the direction of vehicle 100. Forexample, if vehicle 100 configured for use on a road, such as a car ortruck, the steering system may include components to control the angleof wheels to turn the vehicle. Signaling system 166 may be used bycomputing device 110 in order to signal the vehicle's intent to otherdrivers or vehicles, for example, by lighting turn signals or brakelights when needed.

Navigation system 168 may be used by computing device 110 in order todetermine and follow a route to a location. In this regard, thenavigation system 168 and/or data 132 may store map information, e.g.,highly detailed maps identifying the shape and elevation of roadways,lane markers, intersections, crosswalks, speed limits, traffic signallights, buildings, signs, real time traffic information, vegetation, orother such objects and information. The lane markers may includefeatures such as solid or broken double or single lane lines, solid orbroken lane lines, reflectors, etc. A given lane may be associated withleft and right lane lines or other lane markers that define the boundaryof the lane. Thus, most lanes may be bounded by a left edge of one laneline and a right edge of another lane line.

FIG. 2 is an example of detailed map information 200 for a section ofroadway including an intersection 202. In this example, the detailed mapinformation 200 includes information identifying the shape, location,and other characteristics of lane lines 210, 212, 214, traffic signallights 220, 222, 224, 226, crosswalks 230, 232, and sidewalks 240. Eachlane may be associated with a rail 250, 252, 254, 256 which indicatesthe direction in which a vehicle should generally travel in therespective lane. For example, a vehicle may follow rail 252 when drivingin the lane between lane lines 210 and 212.

Although the detailed map information is depicted herein as animage-based map, the map information need not be entirely image based(for example, raster). For example, the detailed map information mayinclude one or more roadgraphs or graph networks of information such asroads, lanes, intersections, and the connections between these features.Each feature may be stored as graph data and may be associated withinformation such as a geographic location and whether or not it islinked to other related features, for example, a stop sign may be linkedto a road and an intersection, etc. In some examples, the associateddata may include grid-based indices of a roadgraph to allow forefficient lookup of certain roadgraph features.

FIGS. 3A-3D are examples of external views of vehicle 100. As can beseen, vehicle 100 includes many features of a typical vehicle such asheadlights 302, windshield 303, taillights/turn signal lights 304, rearwindshield 305, doors 306, side view mirrors 308, tires and wheels 310,and turn signal/parking lights 312. Headlights 302, taillights/turnsignal lights 304, and turn signal/parking lights 312 may be associatedthe signaling system 166. Light bar 307 may also be associated with thesignaling system 166.

FIG. 4 is an example internal view of vehicle 100 through the opening ofdoor 306. In this example, there are two seats 402 for passengers with aconsole 404 between them. Directly in ahead of the seats 402 is adashboard configuration 406 having a storage bin area 408 and theinternal electronic display 152. As can be readily seen, vehicle 100does not include a steering wheel, gas (acceleration) pedal, or brake(deceleration) pedal which would allow for a semiautonomous or manualdriving mode where a passenger would directly control the steering,acceleration and/or deceleration of the vehicle via the drivetrain.Rather, as described in further detail below, user input is limited to amicrophone of the user input 150 (not shown), features of the console404, and, if available, wireless network connections. In this regard,internal electronic display 152 may merely provide information to thepassenger and need not include a touch screen or other interface foruser input. In other embodiments, the internal electronic display 152may include a touch screen or other user input device for enteringinformation by a passenger such as a destination, etc. Similarly, thevehicle may include a steering, acceleration and braking input that apassenger can use to control the vehicle in a manual or semi-autonomousdriving mode.

FIG. 5 is a top down view of the console 404. Console 404 includesvarious buttons for controlling features of vehicle 100. For example,console 404 includes buttons that may be found in a typical vehicle suchas buttons 502 for locking and unlocking the doors 306, buttons 504 forraising or lowering the windows of doors 306, buttons 506 for turning oninternal lights of the vehicle, buttons 508 for controlling a heatingfunction of seats 402, as well as buttons 510 for controlling the volumeof speakers 154.

In addition, console 404 also includes buttons 511 for initiatingcommunication with a remote concierge via a wireless network connectionif available. Buttons 512 and 514 may also be a part of user input 150and in this regard, allow a passenger to communicate with computer 110,for example, to initiate or end a trip in the vehicle. In this regard,button 512 may act as an emergency stopping button that, when pushed,causes vehicle 100 to stop in a short amount of time. Because thepassenger does not have direct control of the acceleration ordeceleration of vehicle 100 by way of a gas or brake pedal, button 512may be an emergency stop button that is critical to allowing a passengerto feel safe and act quickly in case of an immediate emergency.

Button 514 may be a multi-function button. For example, button 514 mayhave three different states. In the first state, button 514 may beinactive, that is, if pressed, the vehicle's computer 110 would notrespond by taking any particular action with regard to controlling themovement of the vehicle. In the second state, when the vehicle is readyto begin a trip, the button 514 may change to a “GO” button which apassenger uses to initiate a trip to a destination or drop off location.Once vehicle 100 is moving, button 514 may change to a third state,where the button 514 is a “PULL OVER” button which a passenger users toinitiate a non-emergency stop. In this regard, computer 110 may respondby determining a reasonable place to pull the vehicle over, rather thancoming to a more sudden stop as with the emergency stop button 512.

Thus, passenger communication with computer 110 for navigation purposesmay be limited to button 514, emergency stopping button 512, a shortrange wireless communication system (such as Bluetooth LE) with thepassenger's client computing device, and by sending information from thepassenger's client computing device to a remote server which then relaysthat information to the vehicle's computer. In some examples, apassenger may provide information to the vehicle's computer 110 viavoice commands though the microphone as discussed above. In addition,however, the passenger may communicate with the concierge via a phonecall, an application on the passenger's client computing device, amicrophone, and/or the concierge button 511 and in turn, the conciergemay provide instructions control certain aspects of a vehicle via aconcierge work station.

The one or more computing devices 110 of vehicle 100 may also receive ortransfer information to and from other computing devices. FIGS. 6 and 7are pictorial and functional diagrams, respectively, of an examplesystem 600 that includes a plurality of computing devices 610, 620, 630,640 and a storage system 650 connected via a network 660. System 600also includes vehicle 100, and vehicle 100A which may be configuredsimilarly to vehicle 100. Although only a few vehicles and computingdevices are depicted for simplicity, a typical system may includesignificantly more.

As shown in FIG. 6 , each of computing devices 610, 620, 630, 640 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto one or more processors 120, memory 130, data 132, and instructions134 of computing device 110.

The network 660, and intervening nodes, may include variousconfigurations and protocols including short range communicationprotocols such as Bluetooth, Bluetooth LE, the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, WiFi and HTTP, and various combinations of theforegoing. Such communication may be facilitated by any device capableof transmitting data to and from other computing devices, such as modemsand wireless interfaces.

In one example, one or more computing devices 110 may include a serverhaving a plurality of computing devices, e.g., a load balanced serverfarm, that exchange information with different nodes of a network forthe purpose of receiving, processing and transmitting the data to andfrom other computing devices. For instance, one or more computingdevices 210 may include one or more server computing devices that arecapable of communicating with one or more computing devices 110 ofvehicle 100 or a similar computing device of vehicle 100A as well asclient computing devices 620, 630, 640 via the network 660. For example,vehicles 100 and 100A may be a part of a fleet of vehicles that can bedispatched by server computing devices to various locations. In thisregard, the vehicles of the fleet may periodically send the servercomputing devices location information provided by the vehicle'srespective positioning systems and the one or more server computingdevices may track the locations of the vehicles.

In addition, server computing devices 610 may use network 660 totransmit and present information to a user, such as user 622, 632, 642on a display, such as displays 624, 634, 644 of computing devices 620,630, 640. In this regard, computing devices 620, 630, 640 may beconsidered client computing devices.

As shown in FIG. 7 , each client computing device 620, 630, 640 may be apersonal computing device intended for use by a user 622, 632, 642, andhave all of the components normally used in connection with a personalcomputing device including a one or more processors (e.g., a centralprocessing unit (CPU)), memory (e.g., RAM and internal hard drives)storing data and instructions, a display such as displays 624, 634, 644(e.g., a monitor having a screen, a touch-screen, a projector, atelevision, or other device that is operable to display information),and user input devices 626, 636, 646 (e.g., a mouse, keyboard,touch-screen or microphone). The client computing devices may alsoinclude a camera for recording video streams, speakers, a networkinterface device, and all of the components used for connecting theseelements to one another.

Although the client computing devices 620, 630, and 640 may eachcomprise a full-sized personal computing device, they may alternativelycomprise mobile computing devices capable of wirelessly exchanging datawith a server over a network such as the Internet. By way of exampleonly, client computing device 620 may be a mobile phone or a device suchas a wireless-enabled PDA, a tablet PC, a wearable computing device orsystem, or a netbook that is capable of obtaining information via theInternet or other networks. In another example, client computing device630 may be a wearable computing system, shown as a head-mountedcomputing system in FIG. 7 . As an example the user may inputinformation using a small keyboard, a keypad, microphone, using visualsignals with a camera, or a touch screen.

In some examples, client computing device 640 may be concierge workstation used by an administrator to provide concierge services to userssuch as users 622 and 632. For example, a concierge 642 may use theconcierge work station 640 to communicate via a telephone call or audioconnection with users through their respective client computing devicesor vehicles 100 or 100A in order to ensure the safe operation ofvehicles 100 and 100A and the safety of the users as described infurther detail below. Although only a single concierge work station 640is shown in FIGS. 6 and 7 , any number of such work stations may beincluded in a typical system.

Storage system 650 may store various types of information as describedin more detail below. This information may be retrieved or otherwiseaccessed by a server computing device, such as one or more servercomputing devices 610, in order to perform some or all of the featuresdescribed herein. For example, the information may include user accountinformation such as credentials (e.g., a user name and password as inthe case of a traditional single-factor authentication as well as othertypes of credentials typically used in multi-factor authentications suchas random identifiers, biometrics, etc.) that can be used to identify auser to the one or more server computing devices. The user accountinformation may also include personal information such as the user'sname, contact information, identifying information of the user's clientcomputing device (or devices if multiple devices are used with the sameuser account), as well as one or more unique signals for the user.

The storage system 650 may also store routing data for generating andevaluating routes between locations. For example, the routinginformation may be used to estimate how long it would take a vehicle ata first location to reach a second location. In this regard, the routinginformation may include map information, not necessarily as particularas the detailed map information described above, but including roads, aswell as information about those road such as direction (one way, twoway, etc.), orientation (North, South, etc.), speed limits, as well astraffic information identifying expected traffic conditions, etc.

As with memory 130, storage system 250 can be of any type ofcomputerized storage capable of storing information accessible by theserver computing devices 610, such as a hard-drive, memory card, ROM,RAM, DVD, CD-ROM, write-capable, and read-only memories. In addition,storage system 650 may include a distributed storage system where datais stored on a plurality of different storage devices which may bephysically located at the same or different geographic locations.Storage system 650 may be connected to the computing devices via thenetwork 660 as shown in FIG. 6 and/or may be directly connected to orincorporated into any of the computing devices 110, 610, 620, 630, 640,etc.

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted.

In one aspect, a user may download an application for requesting avehicle to a client computing device. For example, users 622 and 632 maydownload the application via a link in an email, directly from awebsite, or an application store to client computing devices 620 and630. For example, client computing device may transmit a request for theapplication over the network, for example, to one or more servercomputing devices 610, and in response, receive the application. Theapplication may be installed locally at the client computing device.

The user may then use his or her client computing device to access theapplication and request a vehicle. As an example, a user such as user632 may use client computing device 630 to send a request to one or moreserver computing devices 610 for a vehicle. The request may includeinformation identifying a pickup location or area and/or a destinationlocation or area. As an example, such location may be identified bystreet addresses, location coordinates, points of interest, etc. Inresponse the one or more server computing devices 610 may identify anddispatch, for example based on availability and location, a vehicle tothe pickup location. This dispatching may involve sending information tothe vehicle identifying the user (and/or the user's client device) inorder to assign the vehicle to the user (and/or the user's clientcomputing device), the pickup location, and the destination location orarea.

Once the vehicle 100 receives the information dispatching the vehicle,the vehicle's one or more computing devices 110 may maneuver the vehicleto the pickup location using the various features described above. Asthe vehicle approaches the user's client device, the vehicle's computermay authenticate the user's client device and also the user. When theuser is authenticated, the vehicle's computing devices may automaticallyunlock the vehicle's doors and allow the user to enter the vehicle. Thevehicle's one or more computing devices 110 may also display a welcomescreen on the internal electronic display 152. This welcome screen mayprovide instructions to the user (now a passenger) on how to use thevehicle. For example, the instructions may include requesting that thepassenger shut the doors 306 if needed and buckle his or her seatbelt.Sensors associated with the seats, doors, and seatbelts may be used todetermine if the passenger has complied. Once the passenger has compliedwith the instructions, he or she may press or otherwise activate button514. In response, the computer 110 may initiate the necessary systems tocontrol the vehicle autonomously along a route to the destinationlocation.

As noted above, a vehicle's one or more computing devices may maneuverthe vehicle using the various systems described above. For example, FIG.8 depicts a section of roadway 800 including an intersection 808.Vehicle 100 is approaching intersection 808 and may be controlled, forexample by one or more one or more computing devices 110 in anautonomous driving mode as described above. In this example,intersection 808 corresponds to the intersection 202 of the detailed mapinformation 200, and vehicle is generally following rail 250 in order tofollow a route towards the destination (both not shown in FIG. 8 ). Inthis example, lane lines 810, 812, and 814 correspond to the shape,location, and other characteristics of lane lines 210, 212, and 214,respectively. Similarly, crosswalks 830 and 832 correspond to the shape,location, and other characteristics of crosswalks 230 and 232,respectively, sidewalks 840 correspond to sidewalks 240, and trafficsignal lights 822, 824, and 826 correspond to the shape, location, andother characteristics of traffic signal lights 222, 224 and 226.

As the vehicle is maneuvered to the destination, the vehicle'sperception system 172 may detect and identify objects in the vehicle'senvironment. For instance, the vehicle's computing devices 110 maydetect and identify lane lines 810, 812, and 814, crosswalks 830 and832, sidewalks 840, and traffic signal lights 822, 824, and 826. Inaddition to these “static” features, the vehicle's perception system mayalso detect, track, and identify various other objects such as vehicles850-858 and pedestrians 860, 862. In other words, the perception system172 may determine the general shape and orientation as well as speed ofthese objects by observing these objects over a brief period of time.

When the vehicle operates in a semiautonomous or fully autonomous mode,the vehicle's computers may not only detect these objects, but alsorespond or react to them. For instance, this can include yielding to anobject, stopping, maneuvering around an object, etc. Returning to theexample of FIG. 8 , vehicle 100 may need to yield to group ofpedestrians 860 in crosswalk 832 as well as vehicle 852 which is passingthrough intersection 808 in the direction of pedestrians 860 in order tocontinue its route along rail 250 safely. In addition, the vehicle mayhave to respond to a red light at traffic signal light 826, but at thesame time, may be permitted (according to the traffic laws which governthe intersection defined in the detailed map information 200) to make aright hand turn during the red light after properly stopping andyielding to other traffic. While the vehicle's computing devices 110 maydetect and identify vehicles 850 and 854-858 as well as pedestrian 862,the vehicle's computing devices 110 may determine that no change to thevehicle's current actions is likely to be needed in the immediate future(such as the next few seconds or more or less). In other words, it isunlikely given the location, orientation, and speeds of vehicles 850 and854-858 and pedestrian 862 and the route that vehicle 100 is followingalong rail 250, that vehicle 100 would need to take any action in orderto avoid these objects in the immediate future.

An in-vehicle display can display information about how the vehicle'scomputers plan to react to those objects, or rather, the vehicle'sintent. In this regard, the in-vehicle display can include a scenedisplay for displaying a one or more representative objects and a scenecorresponding to an area around the vehicle and an intent display fordisplaying intent of the vehicle with respect to one or more of theseobjects.

FIG. 9 is an example display 900 which may be presented to an occupantof the vehicle on internal electronic display 152 while the vehicle isbeing maneuvered to the destination. In this example, display 900includes a scene display 910 which displays a scene corresponding tosection of roadway 800 of FIG. 8 . This scene includes a backgroundimage displayed in a background color scheme, here in blues and silvers.The background images corresponds to features of the detailed mapinformation 200 such as lane lines 910, 912, and 914 and crosswalks 930and 932, which correspond to lane lines 210, 212, and 214 and crosswalks230 and 232, respectively. In addition, the display 910 includesrepresentations of various objects 920-928 which correspond to vehicles850, 852, 856, and 858 as well as group of pedestrians 860,respectively. The display 910 also includes a representation 902 ofvehicle 100 and a representation 960 of a path corresponding to rail 250that the vehicle is currently following to reach the destination.

In addition to scene display 910, display 900 also includes an intentdisplay 940 for display the vehicle's intent with regard to various ofthe representations in the scene display 910. For instance, intentdisplay 940 includes a plurality of two-dimensional intent icons 950,952, and 954. Although the term “icon” is used throughout to refer to arepresentations of the vehicle's intent, this term as used herein mayinclude other types of representations, such as photographs, jpegs,pdfs, or other types of images.

In the example of intent display 940, intent icon 950 indicates that thevehicle's computing devices 110 recognize that traffic signal light 826is currently displaying a red (stop) light such that vehicle 100 muststop before intersection 808 and yield to other objects crossingintersection 808 in order to make a lawful right turn as discussedabove. Intent icons 952 and 954 indicate that the vehicle's computingdevices intend to yield to group of pedestrians 860 represented byrepresentations 928 and to vehicle 852 represented by representation922.

The scene and intent displays can be arranged in various ways. Forinstance, the scene display and intent display may be displayed on thesame or different electronic displays of the vehicle. The intent displaymay therefore actually include a sub-area of the scene display such as aside or corner area of the scene display such that the intent displayappears to overlap or is overlaid or superimposed on the scene display.Similarly, the intent display and the scene display may be depicted asdiscrete areas of an electronic display such that there is no overlap oroverlay as shown in FIG. 9 .

FIG. 10 is an example of three different intent displays 1010, 1020, and1030, providing the different information from intent display 950 ofFIG. 9 , but a similar arrangement of two intent icons and one trafficsignal light icon. In the example of intent display 1010, the icons areshown on a black field as if the intent display 1010 were separatelyarranged from intent display 910, such as on another internal electronicdisplay of vehicle 100 or as a discrete portion of display 900. Intentdisplays 1020 and 1030 are displayed as if superimposed over scenedisplay 910 allowing for features of the background scene to bedisplayed “behind” the icons of the intent display. Display 1030 showsthe background features behind the icons with a blur to furtherdistinguish the icons from the background features.

In addition, the scene display and intent display may be distinguishedbased on how the features in these displays are depicted. As an example,the scene of the scene display 910 may be a three-dimensional (3D) imagehaving a depth while the features of the intent display 940 may appearto be flat or two-dimensional (2D). Alternatively, both the scene andintent displays may appear to be in 2D or 3D.

The relationship between a representative object and the vehicle'sintent relative to an object may be presented using colors. As anexample, pedestrian representation 928 shown in the scene display 910may have a yellow ring or other indicator such as a pointer or markerlocated nearby the representation. At the same time, the correspondingintent icon 954, indicating the vehicle's computing devices 110's intentto yield (or simply an acknowledgement that an object has beenidentified as a pedestrian), may be displayed in the same or a similarcolor (here yellow) or with a ring in a same or a similar color (hereyellow) in the intent display 940. This may indicate the vehicle'scomputing devices 110's intent to yield to the group of pedestrians 862in a way in which an occupant of vehicle 100 may quickly and easilyrecognize. Although not shown in the example of FIG. 9 , the intentdisplay 940 may also be used to identify corresponding intent icons forother types of objects such as bicyclists, constructions zone objectslike cones or barriers, etc.

In other examples, objects need not include rings or other markers toidentify corresponding intent icons, but may be shown in a correspondingcolor as the intent icon or a ring of the intent icon. For instance,representation 922 of vehicle 852 is shown in the scene display 910 inpurple to correspond with the purple ring of intent icon 952 in theintent display 940. This may indicate the vehicle's computing devices110's intent to yield to the vehicle 852 in a way in which an occupantof vehicle 100 may quickly and easily recognize. Similarly, turning toFIG. 13 , representation 1310 of pedestrians is shown in silver to blendwith the background color scheme as these pedestrians do not correspondto an intent icon, while representation 1320 of pedestrians is shownhighlighted in yellow to correspond to the yellow ring and color of anintent icon, such as intent icon 954. The corresponding color of therepresentation may be instead of or in addition to a colored ring ormarker (as in the example of scene display 910.

Also, representations of different types of objects can be shown in thesame color and/or with the same color ring or marker in the scenedisplay when the same intent is indicated for those representations inthe intent display. As an example, if the vehicle 100 is yielding toboth a bicyclist and a vehicle, both representations of these objectsmay be shown in purple and/or with a purple ring. However, in somecases, such as in the example of display 900, it may be clearer and morereassuring to the occupant if different intent icons are used fordifferent objects to enable the occupant to understand that thevehicle's computing devices 110 have recognized the differences in howthe vehicle should respond to a group of pedestrians (shown byrepresentation 928) and a vehicle (shown by representation 922). Ofcourse, the colors, shapes, markers, and icons used may be changedaccording to the number of objects and intent or aesthetics of thedisplay.

In addition, to distinguish between intents for different objects of asimilar type, the rings and/or colors for the different intent icons andrepresentations may be displayed in different colors. As an example, aring around one pedestrian may correspond in color to a ring for anintent to yield icon while a ring around another pedestrian maycorrespond in color to a ring for an intent to stop icon (not shown).

Of course, the colors and shapes may be changed according to the numberof objects and intent or aesthetics of the display. For instance, inaddition or alternatively to the examples of yellows and purplesdiscussed above, reds, greens, oranges, and other such colors may alsobe used. In addition, although the example of FIG. 9 depicts the sameshades of yellow and purple between the scene and intent displays,similar hues of the same color (a lighter shade of red and acorresponding darker shade of red, etc.) may also be used to identifycorrespondences between the intent icons in the intent display andrepresentations in the scene display. Similarly, in addition oralternatively to the rings and shapes discussed above, other shapes,such as polygons, etc. may also be used.

In many examples, the vehicle's perception system 172 will detectsignificantly more objects in the vehicle's external environment thanare useful or helpful to display to a passenger or occupant of thevehicle. Objects that are identified must therefore be filtered beforerepresentations of those objects are included in the display 900. Bydoing so, this will reduce the amount of “clutter” in the objectdisplay.

Filtering may be achieved, for example, using various heuristics relatedto the vulnerability of an object (pedestrians and bicyclists havinghigher priority than other objects unless those other objects are doingsomething else interesting), the category of action for the object(moving in the same lane, merging, turning into the lane, etc.), whetherthe vehicle needs to take an action to avoid the object (change lanes,slow down, turn the wheels, etc.), distance from the vehicle, speed ofthe object, etc. For example, representations of pedestrian 860 andvehicle 854 are not included in the scene display 910. In this example,even though pedestrian 860 is detected by vehicle 100, it may be out ofa field of view presented in the scene display 910. Alternatively, ifpedestrian 860 is within the field of view presented in the scenedisplay, a representation of pedestrian 860 may not be displayed in thescene display because the pedestrian is not currently relevant to thevehicle, or rather, the location of pedestrian relative to the vehicle100 is unlikely to affect vehicle 100's route to the destination orrequire some action by the one or more computing devices 110 in order tosafely navigate vehicle 100 along the route. In other words, it isunlikely that the vehicle 100 would need to respond to this pedestrian.In yet another alternative, even where pedestrian is within the field ofview but not currently relevant (as discussed above), a representationof the pedestrian may be included because of the vulnerable nature ofpedestrians. In doing so, this may allow an occupant of the vehicle tofeel more comfortable in that the vehicle is able to recognizepedestrians in the vehicle's environment.

Returning to the example of FIG. 9 , a representation of vehicle 854 maynot be included in the scene display because the location, orientation,and speed of vehicle 854 relative to vehicle 100 is unlikely to affectvehicle 100 as it continues along the route, in other words, vehicle 100is not likely to need to respond to vehicle 854. At the same time,representations of vehicle 856 and 858 are included in the scene display910 because in this example, even though vehicles 856 and 858 are movingtowards intersection 808 and in a different direction from vehicle 100such that vehicle 100 is unlikely to need to respond to vehicles 856 and858, vehicle 100 is actually approaching vehicles 856 and 858. In otherwords, vehicle 100 will be getting closer to these vehicles in the nearfuture, while vehicle 854 will likely continue to move away from vehicle100. Thus, in this situation, representations 924 and 926 may beincluded in the scene display in order to indicate that the vehicle hasdetected these vehicles. Of course, once vehicles 856 and 858 move intointersection 808 and vehicle 100 continues further down rail 250,vehicles 856 and 858 will become less relevant to vehicle 100. At thistime, representations 924 and 926 may fade or disappear from the scenedisplay 910 indicating that vehicles 856 and 858 have become lessrelevant to the vehicle 100.

In addition, certain road features identified the detailed mapinformation, such as construction zones or traffic signal lights, may beomitted from the scene display, but the intent for those features(cautious or slower than usual driving and/or stopping) may still beindicated in the intent display. For example, representations of trafficsignal lights 822, 824, and 826 are not included in the scene display910 in order to reduce the amount of clutter and unnecessary items shownin the display 900. However, at the same time, relevant informationregarding traffic signal lights which are of importance or relevance tothe vehicle 100 given its current position, may be included in theintent display 940, as can be seen from intent icon 950.

In some instances, representations of different types or categories ofobjects can be shown in the scene display using different imagetreatments in order to quickly and easily indicate information about theobjects to a passenger in the vehicle. For example, representations ofstatic objects or background features from the detailed map information200, such as lane lines 910, 912, and 914, crosswalks 930 and 932, orother road features, may be displayed in one color scheme (i.e. allblues or grays), while representations of objects not in the detailedmap information that are relevant to vehicle 100, such as representation922 of vehicle 852, representation 928 of group of pedestrians 862, orbicyclists (not shown) may be shown in different colors in order todistinguish them from the background scene and color scheme.

For instance, representations of moving objects, such as representations922 and 928 to which the vehicle is actually reacting (i.e. changing itsbehavior to slow down, yield, etc.) may be distinguished from thebackground features using different colors. This may indicate to theoccupant that these objects are of relevant for controlling the vehicle100 safely as discussed above. At the same time, moving objects which donot affect the vehicle's behavior may also be shown in the backgroundcolor scheme such as representations 920, 924, and 926 to indicate thatthey are not currently relevant to controlling the vehicle 100 safely.

FIG. 11 is an example view 1100 of a representation 1110 of a vehicle(similar to representation 922) to which vehicle 100 is reacting and arepresentation 1110 similar to representation 920) of a vehicle to whichvehicle is not reacting. In this example, representation 1120 isdisplayed in a first color, here purple which stands out from thebackground color scheme of display 900, and representation 1120 is shownin a different second color, here blue, which blends or corresponds withthe background color scheme of display 900.

FIG. 12 is an example view 1200 of rings 1210 and 1212 which wouldsurround a representation of a pedestrian, bicyclist or other object(similar to representation 928) to which vehicle 100 is reacting andrings 1220 and 1222 of a pedestrian, bicyclist or other object to whichvehicle is not reacting. In this example, rings 1210 and 1212 aredisplayed in a first color, here yellow which stands out from thebackground color scheme of display 900, and rings 1220 and 1222 areshown in a different second color, here silver, which blends orcorresponds with the background color scheme of display 900.

In this regard, as objects become more or less important, theirprominence or relevance to controlling the vehicle 100 can berepresented through changes in the color of the representations of thoseobjects as shown in the scene display. As an example, another vehicle infront of the vehicle may change from a brighter color to a duller colorwhen the other vehicle has turned onto a different roadway. This may bebecause the other vehicle has become less or no longer relevant tocontrolling vehicle 100 and/or the vehicle's computer has determinedthat the other vehicle no longer should be displayed as relevant. Thus,the other vehicle will appear to “fade” into the background of the scenedisplay for example by changing form the representation 1110 to therepresentation 1120 of FIG. 11 . In another example, another vehiclewhich moves into the vehicle's lane in front of the vehicle may changefrom a color of the background scheme to a brighter color as it becomesmore relevant to the vehicle, for example, by changing from therepresentation 1120 to the representation 1110. In this example, theother vehicle goes from being not relevant to currently relevant tocontrolling the vehicle 100.

In another example, turning to FIG. 12 , pedestrian rings 1210 and 1212are used to highlight representations of pedestrians to which thevehicle 100 is responding whereas rings 1220 and 1222 are silver andallow the representations to blend into the background color scheme. Byswitching between these different colored rings, the vehicle's computingdevices 110 can easily convey the relevant or importance of theseobjects to the actions of the vehicle 100. Alternatively, rather thanchanging colors to blend into or stand out from the background colorscheme, the rings may simply disappear or appear.

Similarly, the scene display 910 may show a representation 960 of a pathof the vehicle which also fades to the background color scheme when thevehicle has come to a stop or is temporarily yielding to an object. Thepath may correspond to rail 250 or a portion of a route to thedestination. In this regard, the changing of the color of the path fromone color that stands out from the background scene to another thatblends with the background scene may easily indicate to the occupantthat the stop or yield maneuver is a temporary one and that the vehiclewill resume the route shortly.

In addition, specific types of the moving objects may be represented inthe scene display in different ways. As an example, more vulnerabletypes of objects, such as pedestrians and bicyclists, may be shown usinglaser point clouds, as in the example of representation 920 of FIG. 9 .Point clouds allow an occupant to potentially pick up on details of apedestrian or bicyclist such as what direction they are facing ormoving. Other less vulnerable objects, such as vehicles, may berepresented using a more abstract representation such as a box, tray orstylized cuboid as in the examples of representations 920-926. In thisregard, different types of vehicles, such as passenger car 856 andtractor trailer 858, are depicted in the scene display with the sametypes of generic representations 924 and 926. In other words, therepresentations 924 and 926 are the same shape and size relative totheir three-dimensional distance from the representation 902 of vehicle100 in the scene display 910 (i.e. representations of objects locatedfurther from representation 902 of vehicle 100 will appear smaller thanrepresentations of objects located closer to representation 902 ofvehicle 100). In this regard, by viewing the scene display, the occupantwould recognize that the representations 924 and 926 in the scenedisplay 910 correspond to vehicles but not the specific types of thosevehicles. These more generic representations in combination with thepoint cloud representations given an occupant a good sense that thevehicle has identified different objects and can also distinguishbetween vulnerable and less vulnerable objects.

In order to further indicate the relationship between an object in thescene display with an intent icon in the intent display and thedirection of the object relative to the vehicle, additional color cuesmay be used. For instance, as shown in FIG. 9 , a series of rings orcircles may be displayed in the scene display 910 in order to indicatedifferent distances from the representation 902. Sections of thesecircles may be highlighted in order to indicate the direction of anobject. The highlighting may be in a coordinating color as a feature ofthe object that is colored to coordinate with an intent icon. Forexample, section 970 is highlighted in yellow to indicate the directionof group of pedestrians 860 represented by representations 928 relativeto the representation 902 of vehicle 101. This yellow color furtherindicates the relationship of the representation 920 with intent icon954, also shown in yellow. Similarly, section 972 is highlighted inpurple to indicate the direction vehicle 852 represented byrepresentation 922 relative to the representation 902 of vehicle 101.Again, this purple color further indicates the relationship of therepresentation 922 with intent icon 952, also shown in purple. Inaddition, the tips or narrow ends of these sections are oriented towardsthe representation 902 and the wide ends are oriented towardsrepresentations 922 and 928.

FIG. 14 is an example flow diagram 1400 that may be performed by one ormore of the computing devices 110. In this example, sensor data isreceived from a sensor configured to detect objects in an externalenvironment of a vehicle at block 1410. An object in the vehicle'senvironment is identified at block 1420. It is determined that thevehicle must yield to the identified object in order to safely maneuverthe vehicle to a destination at block 1430. A representation of theobject is generated for a scene display depicting the externalenvironment of the vehicle at block 1440. The representation of theobject includes a feature having a first color. An intent indicator thatindicates the determination that the vehicle must yield to theidentified object is generated for an intent display depicting theintent of the vehicle with respect to the external environment of thevehicle at block 1450. The intent indicator includes a feature havingthe first color. The representation of the object and intent indicatorare provided for display to an occupant of the vehicle in order toindicate to the occupant a relationship between the representation ofthe object and the intent indicator through the first color at block1460.

FIG. 15 is an example flow diagram 1500 that may be performed by one ormore of the computing devices 110. In this example, sensor data isreceived from a sensor configured to detect objects in an externalenvironment of a vehicle at block 1510. A first object and a secondobject are identified in the vehicle's environment from the sensor dataat block 1520. It is determined that the first object is relevant tosafely maneuvering the vehicle to a destination and that the secondobject is not currently relevant to safely maneuvering the vehicle tothe destination at block 1530. A representation of the first object isgenerated for a scene display depicting the external environment of thevehicle at block 1540. The representation of the first object includes afeature having a first color scheme. A representation of the secondobject is generated for the scene display at block 1550. Therepresentation of the second object has a second color scheme. The firstcolor scheme is configured to stand out from the second color scheme. Athree-dimensional background scene is generated in the second colorscheme using detailed map information corresponding to the externalenvironment of the vehicle at block 1560. The representation of thefirst object, the representation of the second object, and thebackground scene are provided for display to an occupant of the vehiclein order to indicate to the occupant that first object is relevant tosafely maneuvering the vehicle to a destination and that the secondobject is not currently relevant to safely maneuvering the vehicle tothe destination.

FIG. 16 is an example flow diagram 1600 that may be performed by one ormore of the computing devices 110. In this example, sensor data isreceived from a sensor configured to detect objects in an externalenvironment of a vehicle at block 1610. A first object of a first typeand a second object of a second type are identified in the vehicle'senvironment from the sensor data at block 1620. The first type and thesecond type are different. A representation of the first object isgenerated for a scene display depicting the external environment of thevehicle based on the first type at block 1630. The representation of thefirst object includes a display of at least a portion of the sensor datacorresponding to the object and indicates an intent of one or more ofthe vehicle's computers with regard to the first object. Arepresentation of the second object for the scene display based on thesecond type is generated at block 1640. The representation of the secondobject is an abstract representation of an object and indicates anintent of the one or more of the vehicle's computers with regard to thesecond object. The representation of the first object and therepresentation of the second object are provided for display on anelectronic display to an occupant of the vehicle in order to indicate tothe occupant that the first object and the second objects are ofdifferent types and that the first object is more vulnerable to animpact with the vehicle than the second object at block 1650.

FIG. 17 is an example flow diagram 1700 that may be performed by one ormore of the computing devices 110. In this example, sensor data isreceived from a sensor configured to detect objects in an externalenvironment of a vehicle at block 1710. A background scene is generatedfor display corresponding to an area of the external environment of thevehicle at block 1720. A set of objects in the area is identified fromthe sensor data at block 1730. The objects of the set of objectscorrespond to one or more of a vehicle, a pedestrian, and a bicyclist.The set of objects is filtered based on a set of heuristics at block1740. A representation of each object of the filtered second set ofobjects is generated at block 1750. Any generated representations andthe background scene are provided for display to an occupant of thevehicle on an internal electronic display of the vehicle at block 1760.The filtering reduces clutter on the internal electronic display.

Although the examples above relate to a particular configuration for theimages depicted on the internal electronic display 152, various otherarrangements and features may be used to increase safety and comfort ofoccupants of the vehicle.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

The invention claimed is:
 1. A method comprising: receiving, by one ormore processors, sensor data from one or more sensor systems configuredto detect objects in an external environment of a vehicle; identifying,by the one or more processors, an object in the vehicle's externalenvironment using the sensor data; generating, by the one or moreprocessors, a representation of the object for a scene display depictingthe external environment of the vehicle, the representation of theobject including an object feature; generating, by the one or moreprocessors, an intent indicator for an intent display depicting anintent of the vehicle with respect to the external environment of thevehicle, the intent indicator including an intent indicator feature;generating a three-dimensional background scene using map informationcorresponding to the external environment of the vehicle; and providing,by the one or more processors, the representation of the object andintent indicator for display to an occupant of the vehicle in order toindicate to the occupant a relationship between the representation ofthe object and the intent indicator, wherein the three-dimensionalbackground scene is provided as a background for the representation ofthe object including the object feature, wherein the intent indicatorwith the intent indicator feature is configured to be overlaid onto thethree-dimensional background scene, and wherein, when the intentindicator is overlaid on the three-dimensional background scene, an areawithin the intent indicator which overlays a portion of the scenedisplay includes a blur effect for the portion in order to distinguishthe intent indicator from the scene display.
 2. The method of claim 1,wherein the representation of the object is provided for display in afirst area of an electronic display corresponding to the scene display,and the intent indicator is provided for display in a second area of anelectronic display distinct from the scene display.
 3. The method ofclaim 1, wherein the representation of the object is provided fordisplay in a first area of an electronic display corresponding to thescene display, the intent indicator is provided for display as anoverlay on the first area of the electronic display over the scenedisplay.
 4. The method of claim 1, wherein the object feature is a firstring, and the intent indicator feature is a second ring.
 5. The methodof claim 1, wherein the object feature is a first ring drawn around adepiction of sensor data corresponding to the object and the intentindicator feature is a second ring drawn around a depiction of intent ofone or more computing devices of the vehicle.
 6. The method of claim 1,wherein the object feature is an abstract depiction of the object andthe intent indicator feature is a ring drawn around a depiction ofintent of one or more computing devices of the vehicle.
 7. The method ofclaim 1, wherein the intent indicator further includes a text describingthe intent of the vehicle with respect to the external environment ofthe vehicle.
 8. The method of claim 1, wherein the intent indicator is atwo-dimensional image.
 9. A system comprising one or more computingdevices having one or more processors configured to: receive sensor datafrom one or more sensor systems configured to detect objects in anexternal environment of a vehicle; identify an object in the vehicle'sexternal environment using the sensor data; generate a representation ofthe object for a scene display depicting the external environment of thevehicle, the representation of the object including an object feature;generate an intent indicator for an intent display depicting an intentof the vehicle with respect to the external environment of the vehicle,the intent indicator including an intent indicator feature; generate athree-dimensional background scene using map information correspondingto the external environment of the vehicle; and provide therepresentation of the object and intent indicator for display to anoccupant of the vehicle in order to indicate to the occupant arelationship between the representation of the object and the intentindicator, wherein the three-dimensional background scene is provided asa background for the representation of the object including the objectfeature, wherein the intent indicator with the intent indicator featureis configured to be overlaid onto the three-dimensional backgroundscene, and wherein, when the intent indicator is overlaid on thethree-dimensional background scene, an area within the intent indicatorwhich overlays a portion of the scene display includes a blur effect forthe portion in order to distinguish the intent indicator from the scenedisplay.
 10. The system of claim 9, wherein the one or more processorsare further configured to: provide the representation of the object fordisplay in a first area of an electronic display corresponding to thescene display, and provide intent indicator in a second area of anelectronic display distinct from the scene display.
 11. The system ofclaim 9, wherein the one or more processors are further configured to;provide the representation of the object for display in a first area ofan electronic display corresponding to the scene display, and providethe intent indicator as an overlay on the first area of the electronicdisplay over the scene display.
 12. The system of claim 9, wherein theobject feature is a first ring, and the intent indicator feature is asecond ring.
 13. The system of claim 9, wherein the object feature is afirst ring drawn around a depiction of sensor data corresponding to theobject and the intent indicator feature is a second ring drawn around adepiction of intent of one or more computing devices of the vehicle. 14.The system of claim 9, wherein the object feature is an abstractdepiction of the object and the intent indicator feature is a ring drawnaround a depiction of intent of one or more computing devices of thevehicle.
 15. The system of claim 9, wherein the intent indicator furtherincludes a text describing the intent of the vehicle with respect to theexternal environment of the vehicle.
 16. The system of claim 9, whereinthe intent indicator is a two-dimensional image.
 17. A non-transitory,tangible computer readable medium on which instructions are stored, theinstructions when executed by one or more processors cause the one ormore processors to perform a method, the method comprising: receivingsensor data from one or more sensor systems configured to detect objectsin an external environment of a vehicle; identifying an object in thevehicle's external environment using the sensor data; generating arepresentation of the object for a scene display depicting the externalenvironment of the vehicle, the representation of the object includingan object feature; generating an intent indicator for an intent displaydepicting an intent of the vehicle with respect to the externalenvironment of the vehicle, the intent indicator including an intentindicator feature; generating a three-dimensional background scene usingmap information corresponding to the external environment of thevehicle; and providing the representation of the object and intentindicator for display to an occupant of the vehicle in order to indicateto the occupant a relationship between the representation of the objectand the intent indicator, wherein the three-dimensional background sceneis provided as a background for the representation of the objectincluding the object feature, wherein the intent indicator with theintent indicator feature is configured to be overlaid onto thethree-dimensional background scene, wherein, when the intent indicatoris overlaid on the three-dimensional background scene, an area withinthe intent indicator which overlays a portion of the scene displayincludes a blur effect for the portion in order to distinguish theintent indicator from the scene display.
 18. The medium of claim 17,wherein the representation of the object is provided for display in afirst area of an electronic display corresponding to the scene display,and the intent indicator is provided for display in a second area of anelectronic display distinct from the scene display.
 19. The medium ofclaim 17, wherein the representation of the object is provided fordisplay in a first area of an electronic display corresponding to thescene display, the intent indicator is provided for display as anoverlay on the first area of the electronic display over the scenedisplay.
 20. The medium of claim 17, wherein the intent indicatorfurther includes a text describing the intent of the vehicle withrespect to the external environment of the vehicle.
 21. The medium ofclaim 17, wherein the intent indicator is a two-dimensional image.